---
title: Set up automation rules
sidebarTitle: Set up automation rules
---

While you can manually sift through and process production logs from your LLM application, it often becomes difficult as your application scales to more users.
LangSmith provides a powerful feature called **Automations** that allow you to trigger certain actions on your trace data.
At a high level, automations are defined by a **filter**, **sampling rate**, and **action**.

Automation rules can trigger actions such as: adding traces to a dataset, adding to an annotation queue, triggering a webhook (e.g. for remote evaluations) or extending data retention. Some examples of automations you can set up:

- Send all traces with negative feedback to an annotation queue for human review
- Send 10% of all traces to an annotation queue for human review to spot check for issues
- Upgrade all traces with errors for extended data retention

<Info>
To configure online evaluations, visit the [online evaluations](/langsmith/online-evaluations) page.
</Info>

## View automation rules

Head to the **Tracing Projects** tab and select a tracing project. To view existing automation rules for that tracing project, click on the **Automations** tab.

![View automation rules](/langsmith/images/view-automation-rules.png)

## Create a rule

![](/langsmith/images/aq-spot-check-rule.gif)

#### 1. Navigate to rule creation

Head to the **Tracing Projects** tab and select a tracing project. Click on **+ New** in the top right corner of the tracing project page, then click on **New Automation**.

#### 2. Name your rule

#### 3. Create a filter

Automation rule filters work the same way as filters applied to traces in the project. For more information on filters, you can refer to [this guide](./filter-traces-in-application)

#### 4. Configure a sampling rate

Configure a sampling rate to control the percentage of filtered runs that trigger the automation action.

You can specify a sampling rate between 0 and 1 for automations. This will control the percent of the filtered runs that are sent to an automation action. For example, if you set the sampling rate to 0.5, then 50% of the traces that pass the filter will be sent to the action.

#### 5. (Optional) Apply rule to past runs

Apply rule to past runs by toggling the **Apply to past runs** and entering a "Backfill from" date. This is only possible upon rule creation. Note: the backfill is processed as a background job, so you will not see the results immediately. In order to track progress of the backfill, you can [view logs for your automations](./rules#view-logs-for-your-automations)

#### 6. Select an action to trigger when the rule is applied.

There are four actions you can take with an automation rule:

- **Add to dataset**: Add the inputs and outputs of the trace to a [dataset](/langsmith/evaluation-concepts#datasets).
- **Add to annotation queue**: Add the trace to an [annotation queue](/langsmith/evaluation-concepts#annotation-queues).
- **Trigger webhook**: Trigger a webhook with the trace data. For more information on webhooks, you can refer to [this guide](./webhooks).
- **Extend data retention**: Extends the data retention period on matching traces that use base retention [(see data retention docs for more details)](/langsmith/administration-overview#data-retention).
  Note that all other rules will also extend data retention on matching traces through the
  auto-upgrade mechanism described in the aforementioned data retention docs,
  but this rule takes no additional action.

## View logs for your automations

Logs allow you to gain confidence that your rules are working as expected. You can view logs for your automations by heading to the **Automations** tab within a tracing project and clicking the **Logs** button for the rule you created.

The logs tab allows you to:

- View all runs processed by a given rule for the time period selected
- If a particular rule execution has triggered an error, you can view the error message by hovering over the error icon
- You can monitor the progress of a backfill job by filtering to the rule's creation timestamp. This is because the backfill starts from when the rule was created.
- Inspect the run that the automation rule applied to using the **View run** button. For rules that add runs as examples to datasets, you can view the example produced.

![Logs_Gif](/langsmith/images/rule-logs.gif)

## Video guide
<iframe
  className="w-full aspect-video rounded-xl"
  src="https://www.youtube.com/embed/z69cBXTJFZ0?si=GBKQ9_muHR1zllLl"
  title="YouTube video player"
  frameBorder="0"
  allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
  allowFullScreen
></iframe>
